Modular-crypto-service #34
Conversation
Signed-off-by: davidliu <[email protected]>
| Following is the original initiative post, translated from Chinese. | ||
| ``` | ||
| The Hyperledger Fabric code project has a very wide range of adoptions in many domestic industries and institutions, and has a very basic role in the application of the blockchain industry. | ||
| According to Chinese policy requirements, it is necessary to support the national secret (GM) encryption algorithm. Many domestic enterprises and Fabric enthusiasts have successively carried out some corresponding national secret (GM) fabric forks with outstanding achievements. However, in this matter, there is serious duplication of working hours and waste of human resources. |
There was a problem hiding this comment.
Can you please explain, or give a formal (English) reference to the bespoken policy? Is this policy only for intra-national networks? I'm asking because most of the world uses the (American) NIST standard - America, Australia, Europe, Africa and Asia.
When a person from China connects to a non chinese website, with overwhelming probability, that website uses the NIST approved algorithms. So, does this policy actually take place? How does the browser of the end user know which protocol to use when connecting to a site?
There was a problem hiding this comment.
We stored a copy of one of original specifications in Financial distributed ledger technology security specification
More recently, there is another related specifications in Financial application of blockchain technology -- Evaluation rules
There might be a blur expression from our original initiative. Policy means some points like:
To promote GM crypto adoption in Critical Industries. This has been mentioned by government from about 10 years ago. 2020 we see the above specifications responding to this policy trend.
Formal English reference does not exist theoretically. But we TWGC are also working on an Fabric Evaluation on the first specs in Tencent doc. It is better formatted to apply google translate and maturity is Beta
About these specification applying scope, both above are limited to 'Finance', in level of 'Recommended'. So it will not apply to all intra-national network or internet. We can see it as an extension from FIPS series when coming to Financial DLT domain.
There was a problem hiding this comment.
The two links you posted are in Chinese, I can't read them. Isn't there an English version somewhere?
About these specification applying scope, both above are limited to 'Finance', in level of 'Recommended'. So it will not apply to all intra-national network or internet. We can see it as an extension from FIPS series when coming to Financial DLT domain.
If it's recommended it means it's not mandatory, therefore since all businesses in China can still use the existing Fabric implementation, what business value does using Chinese crypto have?
There was a problem hiding this comment.
It is required in practical use case. Increasing numbers of financial institution procument will add this specification as requirement to checklist. If it is indeed valueless, we would not have heard about questions and efforts to make a fabric-GM fork. Ignoring this is not prohibited but at least harmful.
Meanwhile I would like to highlight that this RFC does not aim to inject a native GM support in Fabric. We think for further.
@hartm Hart, what is your opinion?
There was a problem hiding this comment.
I'm not a Fabric maintainer (or even contributor, just a free rider), so I don't think my opinion here will matter too much. But I do like the idea of a plugin crypto architecture for Fabric, which I think would be the cleanest way to do this (Fabric maintainers, please feel free to shoot me down if I'm way off), and I'm always happy to spout off an opinion, so I'll go ahead and comment.
There are other applications that a plugin architecture would enable. For instance, I'd like to be able to replace the core signing algorithms in Fabric with pairing-based signatures. We could then use threshold versions of these signature schemes for a number of useful blockchain applications. For instance, this would allow compact endorsements and small "proofs of blockchain state." I have a bunch more applications in mind that I'd be happy to talk about if people are curious that are too long for a comment here. These are obviously not related to government standards.
However, I think you probably want a much more detailed RFC than what is currently present (maybe I'm missing the details?). Can you propose the exact interfaces and functions you want for modular interoperability? I think the Fabric maintainers will potentially be much more receptive if you propose the kind of interface you want. It looks like just 1) a generic hash primitive and 2) a generic signature primitive will be enough. Figuring out what kinds of formats/functions will work best with Fabric will go a long way here.
Finally, doing something like this would be an ambitious project. The more you can do it in bite sized pieces, the better. It will make testing and security analysis much better. So it might be best if your plan was to come up with a generic interface, put your protocols in that interface, and then slowly migrate Fabric.
Anyways, I hope I'm not ruffling any feathers by commenting here. Thanks everyone for your hard work.
There was a problem hiding this comment.
@hartm Thanks for your valuable feedback. Indeed we are hard working to provide more details, and further changes to the RFC proposal will be pushed later.
text/0000-modular-crypto-service.md
Outdated
| We now have 3 streams of China crypto libraries under Hyperledger-TWGC github organization. | ||
| Some of them has also provided successful ane enterprise-proven bccsp implementations. | ||
| But as along as we go deeper into fabric source, we notice implementing bccsp only could not fulfill all of Chinese national crypto specification requirements. | ||
| There are some cypro library usages outside of bccsp, such as communication protocol, X509 format conversion. |
There was a problem hiding this comment.
The Fabric core is very, very tightly coupled with x509 standard:
All communication assumes TLS which uses x509 and doesn't have pluggable algorithms (though, it's possible to fork the TLS library and make your own version of it).
Fabric authentication infrastructure uses the BCCSP MSP which is also heavily based on x509 and doesn't abstract it out in any way.
It's going to be a lot of work to overhaul the entire code base to abstract out x509, and, if we do this, it'll make backporting changes and bug fixes a pain because the code base would diverge too much.
|
Fundamentally, I don't have any issues with the concept, but how something like this is implemented requires careful thought and likely some redesign on the internal Fabric architecture. I do think we need a more simplified version of which algorithms needs to be supported and for which actions. We'll need to go through and figure out all the crypto settings .... especially the ones related to ledger data and transactions as those must be deterministic across releases and capabilities. As an example, for channels, MSPs really drive the crypto which is used for the most part for signing, although the block hashing also is set independent of this. What Fabric really has is two MSP implementations .... and while we call the main one "BCCSP MSP / Fabric MSP", it is really an X509 PKI-based MSP which happens to delegate crypto functions to the current bccsp. So to me, I'd drive this top down from the MSP layer and of course check to see where other crypto settings come from and ensure that we can delegate to registered algorithms where necessary. I'd likely also add an alternative to PKCS11 with a remote interface (http, unix domain sockets, Hashicorp go-plugin, etc) or possibly allow C-based plugins which we can load via dlopen. But we'd want to take a holistic look at plugin architecture in general. Side note: I do not think you need a fork of gRPC. I think you can leverage custom transport credentials to handle TLS with alternative additional ciphers. |
|
I think Gari gets it right here. Well said. |
There was a problem hiding this comment.
Like @mastersingh24, I don't have a problem with the concept but I have a number of practical concerns. What is outlined in the RFC just barely scratches the surface of what is impacted.
Given the scope, I'd suggest breaking this RFC down into multiple proposals.
| ``` | ||
| We now have 3 streams of China crypto libraries under Hyperledger-TWGC github organization. | ||
| Some of them has also provided successful ane enterprise-proven bccsp implementations. | ||
| But as along as we go deeper into fabric source, we notice implementing bccsp only could not fulfill all of Chinese national crypto specification requirements. |
There was a problem hiding this comment.
BCCSP is not an abstraction worth saving and any work in this space should aim to replace it completely with something better.
There was a problem hiding this comment.
Well, I do not dare to say so loudly. :- D. @guoger Jay has asked about this before. Yes, the design looks so Java than golang. But I personally have no idea how is a better practice in golang world.
There was a problem hiding this comment.
For personal understanding here: even if the BCCSP looks like java than golang which still providing us with interface and we can design/change it following design patterns. We can implement particular design pattern via OO concept or else.
text/0000-modular-crypto-service.md
Outdated
| - Fabric uses hash function based on `crypto_config` section in organization/msp level (controlling feature samples: private data) | ||
| - Fabric uses hash algorithm based on `HashingAlgorithm` value in channel level (controlling feature samples: block hash) | ||
|
|
||
| Beside bccsp, a new X509 interface is introduced. Rework of usage of `crypto/X509` includes making `crypto/X509` as an implementaion of new X509 interface. |
There was a problem hiding this comment.
This is too low a level to address the problem and. in my opinion, is not generally feasible.
|
|
||
| To fulfill national crypto standards, security of communication is a must. | ||
| Fabric family replies on classical https and gRPCs communication. So any implementation should firstly have another fork of crypto-complaint https/gRPCs. | ||
| After these forks become mature, incorporate them into individual national crypto fork of fabric. |
There was a problem hiding this comment.
I keep seeing a reference to a fork for all of the "special sauce." So, is the goal to make it easier to fork or to integrate with Fabric?
There was a problem hiding this comment.
Yes, it is our side goal. Make it easier and standardized for developer to have a crypto fabric fork in appropriate way.
There was a problem hiding this comment.
maybe an easy way too fork, but I suppose we should aim at integrate before we have to fork.
text/0000-modular-crypto-service.md
Outdated
|
|
||
| [**WIP**] along with previous feature examples | ||
|
|
||
| crypto/X509 ==> [X509 interface](https://github.com/Hyperledger-TWGC/fabric-gm-plugins/blob/078c5bac196c2c89190b48b9fa05102800a56c34/interfaces.go#L13) |
There was a problem hiding this comment.
On a technical note, I do not support the definition of that interface. It's binding too many aspects of x509 behind an implementation. This is one of the problems with the existing "plug point" definitions of Fabric that we need to avoid as we move forward.
There was a problem hiding this comment.
agree, for any kind of interface, we should kind about, if it is crypto lib level, or fabric level.
ex bccsp.key interface is fabric level, and x509.xxx is crypto lib level.
|
|
||
| **Fabric Family interopt** | ||
| For Fabric surrounding projects such as SDKs and other hyperledger projects with Fabric support, for a specific nation crypto standard Fabric fork. | ||
| developer community still have to make another SDK fork in order to satisfy secured communication. A lot of cross validation is there. |
There was a problem hiding this comment.
This, to me, complicates the ecosystem quite a bit. This really does imply that once Fabric is forked for crypto, each of the SDKs get a fork, the documentation gets a fork, the build processes fork, CI gets a fork, the maintenance process forks, etc.
So, while I support making Fabric more malleable, it's a small piece of the overall solution space.
There was a problem hiding this comment.
@sykesm we have similar concern and discussion of this complexity. Is there any better and smart roadmap per you think that we can switch to?
There was a problem hiding this comment.
For this part, if we just make the bccsp package modular, which means we may don't need to impacts with CI and others. For SDK we will need it supports with a new module for crypto alg implementation...
* add proposal for sdk changes Signed-off-by: SamYuan1990 <[email protected]> * fix up to add java sdk support for tls and a sample refactor Signed-off-by: SamYuan1990 <[email protected]> * adding sample impl for fabric changes Signed-off-by: Sam Yuan <[email protected]>
|
|
||
| ## Detail proposal for sdk changes | ||
|
|
||
| On fabric sdk side, we need align with the changes, so that from client to network able to use same crypto alg/crypto cruve or tls cert... |
There was a problem hiding this comment.
I would personally prefer to see SDK support focused on the Fabric Gateway client API, which is implemented in Go, Node and Java. This API has been designed with a pluggable approach in mind. For example:
- Uses application-provided hashing implementation so any hashing algorithm can easily be used with no SDK changes.
- Uses application-provided signing implementation so any encryption algorithm or implementation can easily be used with no SDK changes.
- Treats client credentials as opaque bytes, so any identity representation can easily be used with no SDK changes.
- Uses application-provided gRPC connection, so the application is free to configure this connection (including transport security) with no SDK changes.
Note that the Fabric Gateway API is focused on business application runtime, including transaction invocation and eventing, not on operational concerns such as identity enrolment with a CA.
It would be good to understand whether the current Fabric Gateway API meets your needs and, if not, what additional requirements you have related to this RFC.
There was a problem hiding this comment.
Hi @bestbeforetoday ,
Yes, we can ref to Gateway API to make changes for SDK side and refresh this part making them up to date, and there are some questions before we update description for this part in this RFC.
Only for java specific
- For my personal understanding, the grpcs changes is better/needed changed from grpc jar packages.
fabric-java-gateway/sdk
grpc.jar
connection level(netty?) we need change here to meet the request
so, at implementation level, we need to change at the connection level to match the "standard request".
sounds like we have two plan:
- Plan A, just recompile netty and replace netty jar packages via mvn/gradle to have a try without updates fabric-java-gateway/grpc.jar if it is possible.
- Plan B, making a customer impl for all the things and use
application-provided gRPC connection
Also there maybe Plan C as with further considering that grpc matches TLS1.3 rfc and too see we can reuse TLS1.3 or not...
With reasons all above, you know that's the reason for now, just focusing on hash and sign logic in current documents.
But here I have a question about application-provided gRPC connection but where should user makes an object as io.grpc.Channel grpcChannel, compile fabric proto file and next?
-
I am agree gateway sdk client is more easy for user for use, but considering for any existing application using sdk, before we announced as duplicates sdk by gateway. I suppose we need considering any existing application in rfc scope?
-
if I am understand correctly, sign, hash, id(as cert import,export)... we are able to customer crypto related impls in java gateway sdk?
Considering other language
- it there any language such as golang/nodejs gateway supposes this kind of sample?
Overall speaking, I am glad to see if the gateway interface you are sharing resolved parts for the questions and I would like to ask @xiaohui249 's help here to double check if the interface you provide can help resolve hash, sign and id.
CC: @davidkhala
:-)
Sam
There was a problem hiding this comment.
The Java client API for Fabric Gateway just requires a gRPC Channel object to be passed in as the gRPC connection to use. This is a very simple abstract class so you could provide your own complete channel implementation that just extends that class if the gRPC ManagedChannel implementation cannot be made to work for you as required. The Fabric Gateway client API should not need any changes.
The Fabric Gateway client API just uses whatever hash and signing implementation you provide, so you can use whatever crypto implementation you want with no changes to the Fabric Gateway client API. At least provided that crypto implementation uses the pattern of hashing a message to create a digest, and then generating signature bytes from that digest. At the server end the usual Fabric mechanisms are used so Fabric Gateway doesn't make it any easier (or more difficult) to provide a server crypto implementation that matches the client implementation.
The Fabric Gateway client API has very similar implementations in Java, Node and Go, so it is just as easy to plug in your own crypto implementation in each language. The gRPC connection details are slightly different as the gRPC implementation in each language are slightly different.
The Go client requires a grpc.ClientConn, which is a struct so not easily replaceable with your own implementation. The Node client requires a gRPC Client object. If we really had to allow alternative gRPC implementations in these languages, we could probably implement some abstraction to make that possible -- in fact the unit tests already make use of similar internal abstractions -- but I suspect a better route might be to enhance the gRPC libraries themselves if they don't currently provide the required capability.
Of course you are free to target existing SDKs too but I would definitely like for the Fabric Gateway client API to be included. Based on my understanding of what you are trying to achieve, it sounds like the Fabric Gateway client API will be much easier to use the way you want than the existing SDKs.
* update with plugin design Signed-off-by: Sam Yuan <[email protected]> * updated to as discussed offline Signed-off-by: Sam Yuan <[email protected]> * update example with real sample Signed-off-by: Sam Yuan <[email protected]> * fix up according to review comments Signed-off-by: Sam Yuan <[email protected]> * update after review offline Signed-off-by: Sam Yuan <[email protected]>
|
Hi All, I found two interested commit in fabric history. mapping to jira tickets with above things, which means in historical, we support fabric BCCSP running in plugin mode and may I know the reason we removed BCCSP plugin mode support? |
|
I replied to @SamYuan1990 in the mailing list, copying here as well... The work item that removed BCCSP plugin support explains the rationale and points to a Go issue about plugin restrictions: https://jira.hyperledger.org/browse/FAB-15338 In a nutshell, nobody was using them and while they sound good on the surface, it was unlikely that anybody would want to use them after understanding the Go restrictions. |
Topics: Who is going to work on this pluggable crypto service?As I mentioned in Dec contributor meetings, from Hyperledger Fabric point of view.
var oidPublicKeyECDSA = asn1.ObjectIdentifier{1, 2, 840, 10045, 2, 1}
var oidNamedCurveSm2 = asn1.ObjectIdentifier{1, 2, 156, 10197, 1, 301}Not everyone can explain the differents as sample code above, but we just need to do things as Asymmetric encryption as below: func sign(sth string){}
func verify(sth string){}
func encrypt(sth string){}
func decrypt(sth string){}
As a summary, let's think about "the pluggable crypto service for any kind of crypto"(Chinese national/ed25519/...). And we hope by this rfc, we can make fabric able to adapt with any kind of crypto in a pluggable way, so that we don't need hard code and recomplie when we swtich from Topic: Size of this item?Language
Channel configAs crypto been used everywhere in any blockchain system with fact that crypto works within same cruve only. Yes, the crypto pluggable should able to work with channel config. MSP and BCCSP together or not?My personal understanding for this is it related with AAA.(authentication, authorization, admin)
TLSFor me current I hope service mesh/side car can help with this kind of problem. Topic: Golang plug in?Before go to details, in general speaking, assume we already defined the interface for
Sorry guys, as time conflicts I missed contributor meeting at Jan 5. I write down comments above with topics on contributor meeting related with this rfc. |
Signed-off-by: Sam Yuan <[email protected]>
* appending context Signed-off-by: Sam Yuan <[email protected]> * adjust with new design and description Signed-off-by: Sam Yuan <[email protected]> * fix up Signed-off-by: Sam Yuan <[email protected]>
| We put the specific crypto package as a part of bccsp package and rebuild fabric with the specific crypto package.(as build tag) | ||
| Full changes here: https://github.com/Hyperledger-TWGC/fabric/tree/bccsp-gm | ||
|
|
||
| Option Three: |
There was a problem hiding this comment.
@denyeart , thanks you pointed out this modular crypto may need take a look at msp interface/package on contributor meeting last year when we do the review.
pre discussed with TWGC member, I had some attempt with bccsp interface and msp interface refactor/decouple idea as 3rd option here.
Please take a look at and have arrange/schedule review on contributor meeting.
|
Hi, everyone. Any updates? |
Signed-off-by: davidliu [email protected]